Skip to content

CASSJAVA-130: Fix Result-Metadata Race Causing Corrupt SKIP_METADATA Row Decoding - #2094

Open
yifan-c wants to merge 2 commits into
apache:4.xfrom
yifan-c:CASSJAVA-130/fix-result-metadata-race
Open

CASSJAVA-130: Fix Result-Metadata Race Causing Corrupt SKIP_METADATA Row Decoding#2094
yifan-c wants to merge 2 commits into
apache:4.xfrom
yifan-c:CASSJAVA-130/fix-result-metadata-race

Conversation

@yifan-c

@yifan-c yifan-c commented Jul 17, 2026

Copy link
Copy Markdown

Patch by Yifan Cai; Reviewed by TBD for CASSJAVA-130

…Row Decoding

Patch by Yifan Cai; Reviewed by TBD for CASSJAVA-130
@yifan-c
yifan-c requested a review from SiyaoIsHiding July 17, 2026 22:05

@tejalchak tejalchak left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid fix. One follow-up question inline around toMessage worth deciding whether to thread the snapshot there too before/after merge.

Comment on lines 304 to 306
logPrefixJoiner.join(this.sessionName, nodeRequestId, currentExecutionIndex),
resultMetadataSnapshot(statement));
Message message = Conversions.toMessage(statement, executionProfile, context);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking at the race condition window here and wanted to get a second pair of eyes ...

While the callback now safely uses the snapshot, toMessage still reads the live metadata state. I'm wondering if a concurrent schema update between capturing the snapshot and the toMessage execution could cause a mismatch (where the request sends the new metadata ID, but the callback decodes using the old snapshot). Should toMessage accept the snapshot to close any chance of discrepancy?

Since ConversionsMetadataRaceTest calls getResultDefinitions directly rather than driving through toMessage, so it might not catch this without some change.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense. We should share the same metadata snapshot in toMessage

Comment on lines +223 to +238
static class ResultMetadata {
private final ByteBuffer resultMetadataId;
private final ColumnDefinitions resultSetDefinitions;

private ResultMetadata(ByteBuffer resultMetadataId, ColumnDefinitions resultSetDefinitions) {
this.resultMetadataId = resultMetadataId;
this.resultSetDefinitions = resultSetDefinitions;
}

ByteBuffer getResultMetadataId() {
return resultMetadataId;
}

ColumnDefinitions getResultSetDefinitions() {
return resultSetDefinitions;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A minor thought: if this branch is strictly Java 17+, we could consider using a record to avoid the boilerplate code
static record ResultMetadata(ByteBuffer resultMetadataId, ColumnDefinitions resultSetDefinitions) {}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the compiler is pinned to java 8. It cannot use the language feature in jdk 17.

@SiyaoIsHiding SiyaoIsHiding left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of having resultMetadata as a field of BoundStatement?
In DefaultBoundStatement constructor, resultMetadata will basically be a snapshot of the preparedStatement.getCurrentResultMetadata() at that moment.
Then at response decode time, as long as rowsMetadata.columnSpecs.isEmpty(), use the one in BountStatement.resultMetadata, otherwise, update the PreparedStatement.resultMetadata.

@yifan-c

yifan-c commented Jul 24, 2026

Copy link
Copy Markdown
Author

What do you think of having resultMetadata as a field of BoundStatement? In DefaultBoundStatement constructor, resultMetadata will basically be a snapshot of the preparedStatement.getCurrentResultMetadata() at that moment. Then at response decode time, as long as rowsMetadata.columnSpecs.isEmpty(), use the one in BountStatement.resultMetadata, otherwise, update the PreparedStatement.resultMetadata.

@SiyaoIsHiding , thanks for the suggestion! Snapshotting on constructor could make the code more clean.

There are three drawbacks:

  1. When the bound statement is reused, after schema change, it is going to send stable metadata ID on every execution.
  2. Snapshotting becomes a hidden contract that every other BoundStatement implementation to follow. Maybe there is no other implementation at all, but BoundStatement is a public interface.
  3. (2) can be addressed with adding the getResultMetadata public API in the BoundStatement. However, public API change requires more discussion, a larger size of decision. For the same reason, the getCurrentResultMetadata is added to DefaultPreparedStatement, not in PreparedStatement.

Therefore, I'd like to stay with the current implementation. Let me know if the above makes sense.

@SiyaoIsHiding

Copy link
Copy Markdown
Contributor

On second thought, what is the race condition that we are trying to fix again?
Is this the current driver's behavior?

  1. SELECT * ... statement is prepared in Cassandra with md5 abc123, and result set (b, c) that hashes to cde456
  2. column a gets added to the table, server updates the result set to (a, b, c) which hashes to fff789
  3. client sends a DefaultBoundStatement for (statementId=abc123, resultId=cde456) and skip_metadata flag
  4. client receives the response from another DefaultBoundStatement, so it updates the PreparedStatement's resultMetadataId to fff789
  5. On the server side, cde456!=fff789, so C* responds with ROWS(..., no_metadata=false, metadata_changed=true, new_metadata_id=fff789,col specs for (a,b,c))
  6. client updates its column specifications again, and decodes the response with (a, b, c).

Under what circumstances does the driver corrupt result metadata?

@yifan-c

yifan-c commented Jul 29, 2026

Copy link
Copy Markdown
Author

On second thought, what is the race condition that we are trying to fix again? Is this the current driver's behavior?

1. SELECT * ... statement is prepared in Cassandra with md5 abc123, and result set (b, c) that hashes to cde456

2. column `a` gets added to the table,  server updates the result set to (a, b, c) which hashes to fff789

3. client sends a DefaultBoundStatement for (statementId=abc123, resultId=cde456) and skip_metadata flag

4. client receives the response from another DefaultBoundStatement, so it updates the PreparedStatement's resultMetadataId to fff789

5. On the server side, cde456!=fff789, so C* responds with ROWS(..., no_metadata=false, metadata_changed=true, new_metadata_id=fff789,col specs for (a,b,c))

6. client updates its column specifications again, and decodes the response with (a, b, c).

Under what circumstances does the driver corrupt result metadata?

Not really, corruption requires two different bound statements of the same prepared statement in flight around a schema change, where the older one gets a SKIP_METADATA reply and its decode is delayed past a newer one's full-metadata reply that updates the shared cache first. The following is the corruption scenario modified from your example.

  1. SELECT * ... prepared as abc123, current result metadata cde456 → (b, c).
  2. Client binds two statements against the same prepared statement — A and B — both while the cache still says cde456/(b,c).
  3. Client sends A's Execute(abc123, resultId=cde456, skip_metadata). Server's current metadata is still cde456 at the moment it processes A (schema hasn't changed yet), so it replies NO_METADATA — just row bytes for (b,c), no column specs, no new id.
  4. Column a is added; server's live metadata for abc123 becomes fff789 → (a,b,c).
  5. Client sends B's Execute(abc123, resultId=cde456, skip_metadata). Now the id mismatches (cde456 != fff789), so server replies with full metadata: column specs (a,b,c), new_metadata_id=fff789.
  6. B's response arrives and is decoded first. Since rowsMetadata.columnSpecs is non-empty, the driver updates the shared PreparedStatement's cached metadata to fff789/(a,b,c).
  7. A's response arrives next. It's the NO_METADATA reply from step 3 — empty columnSpecs, just 2 columns' worth of row bytes laid out as (b,c). The driver decodes it via the 3-column (a,b,c) layout swapped in step 6. The decoder now applies a 3-column definition to 2-columns'-worth of bytes → wrong types/offsets against real data, i.e. corruption (or an exception, depending on type sizes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants